In p we trust (too much). Simulations of typical p-value and effect size behavior in advertising research
(A2019-9521)

AUTHORS

Tim Smits, Institute for Media Studies, KU Leuven; Ben Van Calster, KU Leuven

KEYWORDS

null-hypothesis significance testing; replicability; advertising effects

ABSTRACT

With this simple illustration we aim to demonstrate the most important issues concerning the continued use of p-values and null hypothesis significance testing. We do this based on a set of simulations that pertain to the setting of an advertising researcher investigating a basic effect in a control-versus-experimental between-subjects design. These simulations were performed for different hypothetical true effect sizes and sample sizes and, in a second set of simulations, for some reference values coming from advertising meta-analyses. Important issues demonstrated in these simulations are the high variability of p-values, the variability of effect sizes, and the over-confidence when significant findings occur. These effects are worse for smaller sample sizes. We also focus on the effect of shifting the significance criterion from .05 to .005 as has been suggested recently. We also reflect on how these issues partially contribute to the replicability crisis.